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ABSTRACT 

The  design  of  programs  that  generate  ciuestions  and  exercises 
for  students  is  discussed.  The  task  of  direct  interest  is  the 
generation  of  programming  exercises.  This  is  dealt  with  in  coma 
detail  and  sets  of  such  exercises  produced  by  programmed 
generators  are  presented. 

Several  design  strategies  are  investigated,  starting  with  the 
use  of  grammar- I i ke  generative  mechanisms.  The  design  of 
exercises  is  then  vieued  as  an  assembly  task,  putting  together 
compatible  elements  to  create  an  acceptable  problem-structure, 
accessing  k mu | edge  of  the  task  ^rea  encoded  as  a  semantic  net.  A 
sentence  synthesizer  which  generates  coherent  English  sentences  to 
describe  the  assembled  problems  1;  then  described.  Attention  is 
also  paid  to  the  mechanisms  necessary  for  controlling  such  an 
assembly  process.  Directing  generators  towards  problems  involving 
specified  concepts  and  making  use  of  information  available  about 
the  student's  familiarity  with  a  hierarchy  of  concepts  are  two 
issues  discussed  in  th  i  j  are.'.. 

Other  strategies  discussed  are:  generalizing  ’good'  problems 
to  produce  useful  variants;  use  of  reasoning  about  actions  in  a 
task  area  to  recognize  good  problem  situations;  and  the  design  of 
surface  detail  to  create  problems  having  a  given  abstract 
struc  ture. 
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L_  INTRODUCTION 

The  notion  of  a  generative  CA1  system  has  attracted  some 
at  tent i on  11 ) .  An  important  issue  in  this  area  ss  th°  structure  of 
generators  for  problems  and  exercises,  and  it  is  this  issue  that 
we  concern  ourselves  with  in  the  following  pages.  Design  of 
generators  for  interesting  classes  of  problems  appears  within 
reach  and  is,  of  course,  advantageous.  Among  the  frequently  cited 
advantages  of  generation,  as  against  selection  from  a  long  list, 
are  the  controllability  and  adaptability  that  generation  provides. 
These  features  provide  for  the  creation  o.r  exercises  having 
specified  content  and  posing  a  specified  levei  of  difficulty  to 
suit  the  needs  of  an  irdividual  student  at  a  specific  stage  in 
lea-ning.  And,  of  course,  generative  schemes  provide  for 
potentially  larger  sets  of  examples  to  choose  from,  compared  to 
any  finite  list. 

Developing  a  generative  scheme  forces  analysis  of  problems 
and  th„  task  area  to  a  level  of  detail  that  i=  usually  avoided  in 
making  a  collection  of  exercises.  in  fact,  such  analysis  might 
possibly  lead  to  the  the  definition  of  knowledge  domains 
operationally.  For  instance,  in  classifying  exe-cises  on  the 
basis  of  complexity,  one  could  use  a  response  measure  such  as 
'average  time  taken  for  solving’,  but  a  generative  scheme 
motivates  attempts  to  understand  the  difficulty  posed  by  an 
exercise  in  terms  of  its  structure. 

Another  aspect  of  generative  schemes  is  the  level  at  which 
information  is  available  to  the  system  th=>  handles  the  exercises. 
Uhile  a  collection  of  exercises  couid  be  handled  by  a  more  or  less 
sophisticated  page-turner,  generative  schemes  require  a  fine-grain 


(1)  Uhr,  19b9;  Uttal  et  al,  13735  Uexler,  1970  and  Koffiran,  1972, 
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representation  of  some  knowledge  of  the  task  area  in  order  to 
work.  lie  use  a  semantic  net  in  which  is  encoded  a  variety  of 
information  on  the  tack  area.  In  the  case  of  programming 
exercises,  the  semantic  net  contains  information  on  types  of 
r,  anipulable  objects  such  as  numbers,  symbols,  sets  and  sequences, 
their  attributes,  sets  of  examples  for  each  type  of  object  or 
generative!  procedures  for  creating  examples,  data  structures  in 
terms  of  which  each  type  of  object  may  be  realized,  manipulations 
that  may  be  performed  on  each  type  of  object,  etc..  Dealing  with 
information  in  such  detail  is  essential  if  we  are  ever  to  develop 
systems  that  understand  the  task  area  -  to  generate  answers  to 
exercises  or  to  generate  code  to  recognize  a  variety  of  acceptable 
answers  along  with  the  generation  of  the  exercise,  to  answer 
questions  about  a  specific  exercise  or  about  the  task  area,  and  to 
attempt  to  carry  out  generated  exercises  tc  find  out  their 
character i st ics. 

The  recognition  that  generative  techniques  can  make  good  use 
of  a  fine-grain  representation  of  semantic  information  relevant  to 
a  tack  area  hardly  concludes  the  analysis  of  problem  generation. 
In  fact,  this  recognition  is  more  appropriate  as  the  first  step 
than  as  the  conclusion  in  any  treatment  of  the  subject. 
Development  of  a  problem  generator  in  a  given  area  should  start 
with  the  identification  and  analysis  of  viable  sets  of  problems  in 
that  area,  and  proceed  to  specify  su'iable  structures  for  programs 
that  will  generate  these  sets.  Specific  strategies  for  accessing 
representations  of  knowledge  in  the  process  of  problem  generation 
shou  I  (I  be  spe  It  cut. 

Undoubtedly,  an  understanding  system  uith  a  well  integrated 
store  of  knowledge  of  a  subject  area  is  the  proper  ultimate  form 
for  a  problem  generator.  Such  a  system  would  be  able  to  answer 
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questions  about  the  area,  design  good  problems  in  the  area, 
discuss  them  and  even  solve  them.  The  work  reported  here  does  not 
yet  aim  at  such  a  system.  The  approach  taken  here  is  exploratory 
and  quite  empirical.  The  presentation  is  in  the  form  of  a 
discussion  of  a  sequence  of  simple  strategies  fcr  using  different 
aspects  of  information  about  a  subject  area  for  generating 
non-trivial  problems.  Host  of  the  strategies  discussed  are 
illustrated  with  examples  of  problems  generated  by  programs  based 
on  them.  All  the  examples  presented  are  in  the  area  of 
programming  exercises,  though  the  discussions  are  hopefully 
relevant  to  other  kinds  of  problems  as  well. 


/ 
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Zjl.  ENynERAUVE  SCHEMES  PATTERNED  AFTER  CONTEXT-FREE  GRAMMARS 

The  simplest  generative  scheme  is  an  extension  of  the 
s I ot-and-f i ! I er  scheme,  as  it  is  called  in  linguistics.  In  this 
scneme,  one  has  a  set  of  problem  frames  incorporating  variables. 
There  is  a  specification  of  the  admissible  values  that  these 
variables  may  assume,  either  in  the  form  of  numerical  iimi  s  or  in 
the  form  of  a  set  of  admissible  values  for  each  variable. 

There  are  two  directions  of  development  that  enrich  this 
scheme.  Uhrtl)  reports  one  of  them,  which  is  to  implement 
computationally  defined  relations  between  the  variables.  By 
choosing  ce*tair.  independent  variables  randomly  within  the 
permitted  ranges  or  from  sets  of  admissible  values,  the  others 
could  be  computed  from  tram,  thus  ensuring  compatibility.  The 
original  scheme,  of  course,  does  not  provide  mechanisms  for 
meeting  such  compatibility  requirements  and  forces  the  variables 
to  be  cho'en  independently  of  each  other. 

The  second  direction  of  development  involves  the  use  of  a 
grammar- 1  i ke  scheme.  Unlike  siot-and-f i I ler  schemes,  the 
grammar- I  ike  schemes  are  not  limited  tc  finite-state  languages. 

They  have  been  used  in  the  context  of  problem  generation  for 

synthesizing  sentences  having  phrase  structure.  Koffman  (1972) 

has  reported  the  use  of  probabilistic  grammars  for  the  generation 
of  word  problems.  Simple  programs  that  use  grammar- I i ke  schemes 
to  generate  questions  in  artificial  intelligence  (developed  by 

Newell  and  Robertson)  and  in  cognitive  psychology  (developed  by 
Waterman)  have  been  in  use  at  the  Carnegi e-Mel  ion  University  since 
1371, 
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Fig.l  shows  the  e.,umerative  scheme  underlying  the  program 
which  deals  with  artificial  intelligence.  The  questions  produced 
by  this  program  are  always  grammatically  and  semantically  correct, 
and  most  of  the  questions  are  meaningful .  Not  shown  in  the  scheme 
is  a  probability  assignment  convention  that  allows  certain  choices 
to  be  more  or  less  frequent  than  other  choices  in  the  generative 
procedure,  increasing  the  percentage  of  desirable  questions  in  the 
output.  Figures  2  and  3  show  some  of  the  questions  produced  by 
the  artificial  intelligence  program  and  the  cognitive  psychology 
program  respectively. 

Enumerative  schemes  such  as  this  derive  their  power  from  a 
basic  device  --  classification  of  a  set  of  phrases  into  different 
subsets  of  syntactically  and  semantically  similar  items.  Question 
frames  are  written  incorporating  variables  which  are  substitutable 
by  members  of  appropriate  subsets  of  phrases.  Further,  the 
phrases  are  themselves  generatabie  by  this  process  by  expansions 
of  embedded  variables  into  suitable  sub-phrases,  the  variables  in 
figure  1  are  indicated  by  the  terminal  angle  brackets  ’<’  and  *>’. 
sets  of  phrases  that  are  substitutable  for  the  variables  are 
defined  in  the  lower  half  of  the  figure. 
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<QUEST  IONS>:  (  ("UHAT  IS  "  <ENTI TY>  "?') 

("WHAT  METHOD  DOES  ”  <PROGRAM>  "  USE?") 

("UHAT  TASrS  OCcS  "  <PROGRAn>  "  WORK  ON?") 

("HAS  "  <TASK>  "  BEEN  ACCOMPLISHED  BY  A  PROGRAM?") 

("WHERE  IS  AI  RESEARCH  ON  ■'  <TASK>  "  GOING  ON?") 

("UHAT  IS  THE  CURRENT  ACHIEVEMENT  IN  "  <TASK>  "?") 

("UHAT  HAVE  BEEN  THE  CONTRIBUTIONS  OF  AI  TO  "  <AREA>  "?") 

("UHAT  HAVE  BEEN  THE  CONTRIBUTIONS  PF  "  <AREA>  "  TO  AI?") 

("NAME  A  TASK  THAT  "  <PRGGRAM>  "  OOES  NOT  DO,  BUT  MIGHT  DO  WITH  SOME 
MODIFICATION.") 

("UHY  IS  "  <ENTI TY>  "  SO  CALLED?") 

("UHAT  PROGRAMMING  LANGUAGE  UAS  USED  FOR  "  <PROGRAM>  "?") 
f'HAS  "  <LANGUAGE>  "  BEEN  USEO  TO  ACCOMPLISH  "  <TASK>  "?")  ) 

<ENTITV>:  ( (<PROGRAfi>)  ( cHEURI  ST  I  C>)  (<PHENCiMENA>)  (<CONCEPT>)  . ..) 
<CONCEPT>'  H" A  METHOD”)  ("A  PROBLEM")  ("A  HEURISTIC")  ...) 

<PROGRAM>:  ( (<RECOGNI  ITON-SYSTEMS>)  (<GAME-PLAYERS>)  ( <THEOREM-PROVERS > ) 
( <QUFS  T I  ON - ANSUERERS  > )  ( <GENERAL-PRQBLEM- SOLVERS> )  ... 

( <UNDERST AND  I NG-S YSTEMS> )  ( <DES I GN-SYSTEMS> ) 

(<CONSTRAiNT-SATISFlERS>) ) 

<lHESS-PROGRAMS>:  ( ("NEUELL-SHAU-SIMON  CHESS  PROGRAM") 

("BERNSTEIN’S  CHESS  PROGRAM”)  ("CMU  TECHNOLOGY  CHESS  PROGRAM")  ...) 
<TASK>:  ( (<GAMES>)  (<MANAGEMENT-SCIENCE>)  (<CONCEPT-FORMAT!ON>)  ,..) 
<PROGRAM-FEATURE>:  ( ( <HEUR  I  ST  I  C> )  ("ALPHA-BETA")  ("RECURSION")  ,..) 


F  i  g,  1  Par  t  of  an  Enumerat i ve  Scheme  Patterned  after 
Context  Free  Grammars 
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Qs  UMAT  A1  PROGRAMS  HAVE  BEEN  URiTTEN  IN  ML1SP? 

Q:  WHY  IS  FEATURE  EXTRACTION  SO  CALLEO? 

Q:  HAS  THERE  BEEN  ANY  CONTRIBUTION  OF  AUTOMATA  THEORY  TO  A)? 

Q:  UHAT  NAS  THE  FIRST  AI  EFFORT  ON  CHESS? 

Q:  HAS  SAIL  BEEN  USEO  TO  ACCOfIPLISH  CHECKERS? 

Q:  UHAT  HAVE  BEEN  THE  CONTRIBUTIONS  OF  A I  TO  PHYSICS? 

Q:  HAS  APL  BEEN  USED  TO  ACCOfIPLISH  ASSEMBLY  LINE  BALANCING? 

Q:  UHAT  IS  THE  CURRENT  ACHIEVEMENT  IN  CHESS? 

Q:  UHAT  IS  THE  CURRENT  ACHIEVEMENT  IN  PLANT  LOCATION  SELECTION? 

Q;  UHAT  HAVE  BEEN  THE  CONTRIBUTIONS  OF  INDUSTRIAL  ADMINISTRATION  TO  AI? 
Q:  UHAT  IS  A  HEURISTIC? 

Q;  HAS  SAIL  BEEN  USEO  TO  ACCOMPLISH  RELATIONAL  CONCEPT  FORMATION? 

Q:  HAS  ASSEMBLY  LINE  BALANCING  BEEN  ACCOMPLISHED  BY  A  PROGRAM? 

Q:  HAS  COBOL  BEEN  USEO  TO  ACCOMPLISH  CHESS? 

Q:  HAS  THERE  BEEN  ANY  CONTRIBUTION  OF  PHYSICS  TO  AI? 

Q:  UHAT  IS  HORIZON  PHENOMENA? 

Q:  UHAT  PROGRAMMING  LANGUAGE  UAS  USEO  FOR  REF-ARF? 

Q:  UHAT  UAS  THE  FIRST  A I  EFFORT  ON  RELATIONAL  CONCEPT  FORMATION? 

U:  UHAT  HAVE  BEEN  THE  CONTRIBUTIONS  OF  AI  TO  YOUR  EDUCATION? 

G:  UHAT  METHOO  OOES  GPS  USE? 

Q:  UHAT  TASKS  OOES  FREEMAN’S  OSO  UORK  ON? 

Q:  NAME  SOME  HEURISTICS  USEO  IN  SRI  ROBOT. 

Q:  UHO  DEVELOPED  EXPONENTIAL  GROUTH? 

FJg.2  OyitpM.t  PJt  Hie  Question  Generator  for  Artificial  Intel  I  iaence 
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UHAT  CONTRIBUTION  HAS  LOEHL1N  MADE  TO  THE  5TU0V  OF  COGNITIVE  PSYCHOLOGY? 
UHAT  IS  CYBERNETICS? 

UHAT  ARE  THE  SIMILARITIES  BETWEEN  TREE  STRUCTURES  AND  NETWORKS’ 

UHAT  ARE  THE  SIMILARITIES  BETL'EEN  ALGORITHM  AND  HEURISTIC? 

UHAT  CONTRIBUTION  HAS  ABELSON  MADE  TO  THE  STUDY  OF  COGNITIVE  PSYCHOLOGY? 
UHAT  IS  CONTENT  AOORESSABLE  MEMORY? 

UHAT  IS  AN  EXAMPLE  OF  A  GPS  GOAL? 

UHAT  ISA  MAGIC  SQUARE? 

UHAT  ARE  THE  DIFFERENCES  BETWEEN  SYNTAX  ANO  SEMANTICS? 

DEFINE  MARKOV  PROCESS. 

DESCRIBE  A  FEU  METHODS  THAT  HAVE  BEEN  USED  IN  ARTIFICIAL  INTELLIGENCE. 
UHAT  IS  A  BIT? 

UHAT  ARE  THE  DIFFERENCES  BETWEEN  TEMPLATE  MATCHING  ANO  FEATURE 
EXTRACTION? 

UHAT  IS  AL.OOUS? 

UHAT  CONTRIBUTION  HAS  CHOMSKY  MADE  TO  THE  STUOY  OF  COGNITIVE  PSYCHOLOGY? 
UHAT  IS  PANDEMONIUM? 

UHAT  CONTRIBUTION  HAS  UIENER  MADE  TO  THE  STUDY  OF  COGNITIVE  PSYCHOLOGY? 
UHAT  ARE  THE  SIMILARITIES  BETWEEN  COMPUTER  AND  HUMANS  -  BY  ANALOGY? 
DEFINE  DISCRIMINATION  NET. 

UHAT  ARE  THE  SIMILARITIES  BETWEEN  COMPiLER  AND  INTERPRETER? 

UHAT  ARE  THE  SIMILARITIES  BETUEEN  TEMPLATE  MATCHING  ANO  FEATURE 
EXTRACTION? 


F  i  a ■ 3  Output  qJ  the  Quest i on  Generator  1  or  Coan  j t i ve  Psycho  I oqu 
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Word  categorization  on  a  semantic  as  well  a9  syntactic  basis 
enables  these  programs  to  produce  a  surprisingly  high  percentage 
of  semantically  meaningful  sentences  compared  to  programs  that  are 
based  on  generative  syntax  alone.  Restricting  oneself  to  the 
generative  power  of  ccntext-free  rewrite  rules  has  its  own 
advantages.  The  control  structure  of  the  problem  generator 
becomes  simple  and  straightforward,  making  it  easy  to  construct 
trouble-free  programs.  The  essence  of  these  schemes  is  the  simple 
hierarchical  top-down  control  structure  that  relate  a  set  of 
choices.  For  each  choice  that  is  made,  the  grammar  simply  forces 
the  other  (lower  level)  choices  that  one  is  committed  to  make  as  a 
result.  Each  act  of  choice  transfers  control  to  that  part  of  the 

grammar  where  the  >3 'liable  alternatives  for  lower  level  choices 

are  contained.  On  the  other  hand,  restriction  to  such  a  simple 
program  structure  does  result  in  severe  limitations. 

The  basic  limitation  is  the  absence  of  a  structured 

representation  of  the  task  environment  of  the  xind  found  in 

practically  every  sys^e t  that  performs  a  cognitively  significant 
t3sk  (Carbonell,  197B,  a.  d  Uexler,  1970,  describe  instructional 
systems  incorporating  structured  repr esentat  i  r  1?  of  knowledge). 
General  ly  in  such  systems  associations  are  used  to  impose  an 

object-attribute-value  scheme  on  the  *ask  environment  and  to 
implement  functional  and  relational  mappings.  Computational 
mappings  of  a  functional  or  relational  nature  are  performed  by 
programmed  routines.  This  is  in  contrast  to  the  implicit 

representation  of  knowledge  in  grammar-like  systems  which  provide 
only  two  devices  to  ensure  coherence  in  the  output:  co-occurrence 
of  symbols  in  reurite  rules  which  ensures  co-occurrence  of 

comp  I  ementn- y  parts  of  the  problem  in  the  output;  and  set 

membership  -ihich  enables  semantically  and  syntactically  equivalent 
items  to  ba  distributed  ir  e  output  <n  an  identical  manner. 
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-ven  the  use  of  variables  for  holding  data  and  control 
information  is  not  possible.  This  means  for  instance,  that  one 
Cannot  use  a  variable  to  hold  on  to  the  name  of  the  data-structure 
that  one  has  chosen  to  refer  to  in  a  problem  being  generated,  so 
that  i  t  can  be  used  repeatedly,  instead  one  requires  one  set  of 
ruies  for  generating,  say,  sorting  exercises  on  lists  and  a 
similar  but  different  set  of  rules  for  sorting  exercises  on 
a.  rays,  unless  of  course  there  is  a  single  reference  to  the  nature 
of  the  data-structure. 

Finally,  syntactical  and  semantic  similarities  of  phrases  are 
clumped  together  while  classifying  these  phrases  into  subsets. 
The  fact  that  a  particular  set  of  phrases  refers  to  investigators 
in  a  particular  field  nas  to  be  remembered  along  uith  the 
syntactic  ’nformation  that  all  entries  in  this  subset  are  singular 
in  number,  and  that  in  addition  they  are  all  only  last  names. 

Achievements  of  investigators  expressed  in  the  past  tense  uive  to 
be  segregated  from  their  activities  expressed  in  other  tenses. 
Flexibility  in  reference  can  be  obtained  only  by  entering 
grammatical  variants  of  the  same  phrase  in  different  subsets. 

Summing  up,  it  appears  that  the  initial  advantages  of  a 

grammar-like  scheme  are  outweighed  by  the  restrictions  it  imposes 
on  the  development  of  a  problem  generator.  It  becomes  necessary 
to  recognize  and  separate  two  concepts  implicit  in  such  schemes, 
one  being  more  general  than  the  other.  The  first  is  the  concejt 

of  generative  syntax  with  its  power  to  generate  well-formed 

sentences  in  natural  language  or  in  mathematical  notation.  The 
second  in  the  concept  of  an  elegant  and  economical  principle  of 
p  3gram  organization  -  that  of  a  top-down  enumerative  scheme  for 
generating  a  recursively  defined  set  of  structures.  It  is 
obviously  desirable  to  employ  the  latter  concept  whenever 


possible,  without  committing  oneself  to  the  formalization  of  all 
the  machinery  required  for  problem  generation  in  the  form  of 
generative  syntax.  Steps  in  this  direction  are  particularly 
important  if  we  visualize  CAI  syst 'ms  having  access  to  detailed 
representations  of  knowledge  of  task  areas.  Whatever  form  such 
knowledge  is  going  to  be  in  -  semantic  nets,  simulation  models  or 
understanding  systems  -  it  certainly  is  not  going  to  be  all 
syntax. 
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Si-  GENERATION  QF  A  PROBLEM  as  IH£  assembly  qf  a  structure 


3 . 1  Assemb I u  of  Prob I ems 


Ue  can  view  the  creation  of  exercises  as  a  task  in  design. 
Given  the  function  of  a  desired  object,  hew  does  one  decide  its 
structure?  How  does  one  rscog.  ize  the  system  ot  design  decisions 
necessary  in  a  given  case?  How  does  one  encode  decisions  made  in 
the  earlier  stages  of  the  process  so  that  their  implications  for 
later  decisions  are  readily  computable?  in  this  section  ue  will 
deal  with  the  generation  of  a  basic  kind  of  programming  problem  in 
these  terms.  These  problems  exercise  the  student’s  skills  in 
handling  arrays  and  in  organizing  uata  in  the  form  of  sets  and 
sequences  of  numbers  and  symbols.  A  typical  hand  generated 
problem  of  this  type  (1)  is: 

YOU  ARE  GIVEN  THE  ARRAYS  I  ARRAY [1: 180]  AND  MARRAY [1 : 100) .  FIND 
ALL  SETS  OF  NUMBERS  IN  I  ARRAY,  NO  MATTER  WHAT  THEIR  ORIGINAL 
POSITION,  WHICH  CAN  BE  PUT  IN  COUNTING  ORDER  (I.E.  EVERY  ELEMENT 
IN  A  SET,  EXCEPTING  THE  FIRST  AND  THE  LAST,  SHOULD  HAVE  ITS 
SUCCESSOR  AND  PREDECESSOR  IN  THE  SAME  SET).  MARK  THE  FIRST  SET  BY 
PUTTING  l’S  IN  THE  CORRESPONDING  POSITIONS  IN  MARRAY,  THE  SECOND 
SET  BY  PUTTING  2'S  in  the  CORRESPONDING  POSITIONS,  AND  SO  ON.  PUT 
ZEROES  IN  THE  POSITIONS  OF  ELEMENTS  WHICH  DO  NOT  BELONG  TO  ANY 
SETS. 


Consider  the  task  of  a  program  that  has  to  generate  problems 
in  this  form.  A  series  of  decisions  have  to  be  made,  some  of  them 
independent  and  some  contingent  on  the  ethers.  Analysis  of  a 
number  of  such  problems  shows  that  it  is  possible  to  represent  in 
a  compact  manner  the  basis  Tor  making  these  decisions  for 


(1)  Courtesy  of  Ruven  Brooks 
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producing  reasonably  large  subsets  of  the  studied  problems.  It  is 
possible  in  twese  cases  to  find  a  common  form  in  the  structures  of 
the  .ndividual  members  of  the  problem  set.  For  example,  the 
problems(l)  of  Fig. 4  f  „vc  a  common  structure  outlined  in  Fig. 5. 
The  problems  in  F;g.4  as  well  as  those  in  Fig. 6  were  generated  by 
programs  having  the  simple  control  structure  discussed  in  the  last 
paragraph  of  Section  2.  Use  of  +his  top-down  generative  structure 
is  based  on  the  recognition  that  the  problem  to  be  constructed  has 
a  basically  tree- 1  ike  structure.  The  highest  levei  of  the  program 
therefore  deals  with  the  Inking  up  of  the  major  sub-structures  of 
the  tree  uhich  are  assumed  to  exist.  Lower  level  routines  are 
then  defined  for  creating  these  smaller  structures.  The 
parcelling  out  of  the  assembly  task  C3n  be  spread  over  several 
levels,  the  rout:nes  at  the  terminal  nodes  selecting  and  creating 
basic  elements  of  the  prcblem,  while  ine  higKsr  level  routines 
assemble  parts  created  at  lower  levels  into  bigger  structures. 


(l)Generated  by  a  program  to  be  described  shortly. 


1G 


DEFINE  AN  ARRAY  1(200)  AND  REAO  NUMBERS  INTO  THIS  ARRAY  FROM  THE 
FILE  FOR01.GEN.  CHOOSE  A  SUITABLE  FORMAT  FOR  THE  READ  STATEMENT 
BY  FIRST  TYPING  CUT  FOR01.GEN  ANO  EXAMINING  IT.  EXAMINE  THE 
ELEMENTS  OF  L  ANO  IDENTIFY  THE  MAXIMAL  SET  EACH  ELEMENT  OF  UHICH 
IS  A  PRIME.  SORT  THE  E:EMFNrs  OF  THE  SELECTED  SET  IN  NUMERICAL 
ORDER.  PLACE  THIS  ORDERED  SEQUENCE  OF  NUMBERS  AT  THE  BEGINNING  OF 
L,  IN  DECREASING  NUMERICAL  ORDER.  P!  ACE  A  ZERO  IN  ALL  THE 
LOCATIONS  OF  L  UHICH  DO  NOT  CDNTAIf!  ONE  OF  THE  FINALLY  SELECTED 
NUMBERS.  DRINT  OUT  THE  FINALLY  SELECTED  NUMBERS. 


DEFINE  AN  ARRAY  L (30,30)  ANO  READ  NUMBERS  INTO  THIS  ARRAY  FROM  THE 
FILE  FOR0I.GEN.  CHOOSE  A  SUITABLE  FORMAT  FOR  THE  READ  STATEMENT 
BY  FIRST  TYPING  OUT  FOR01.GEN  ANO  EXAMINING  IT.  EXAMINE  ALL  THE 
ROUS  OF  L  ANO  IN  EACH  ROU.  IDENTIFY  ALL  SEQUENCES  OF  7  OR  MORE 
NUMBERS  HAVING  ELEMENTS  EACH  ONE  OF  UHICH  IS  A  NUMBER  GREATER  THAN 
4.  FOOL  THE  ELEMENTS  OF  ALL  THESE  SEQUENCES  ANO  SORT  THEM  IN 
NUMERICAL  ORDER.  PLACE  THE  38  LARGEST  ELEMENTS  ALONC  THE  LOL'EST 
ROU  IN  NUMERICALLY  INCREASING  DROER  FROM  LEFT  TO  RIGHT.  PLACE  A 
ZERO  IN  ALL  THE  LOCATIONS  OF  L  UHICH  DO  NOT  CONTAIN  ONE  OF  THE 
FINALLY  SELECTEO  NUMBERS.  PRINT  OUT  THE  FINALLY  SELECTED  NUMBERS. 


DEFINE  AN  ARRAY  L (303)  AND  ASSUME  THAT  THIS  ARRAY  IS  FILLED  UITH 
NUMBERS  IN  THE  RANGE  FROM  -1000  TQ  1003.  EXAMINE  THE  ELEMENTS  ClF 
L  ANO  IDENTIFY  ALL  SEQUENCES  OF  3  OR  MORE  NUMBERS  HAVING  ELEMENTS 
THAT  ARE  ALTERNATELY  A  CUBE  ANO  A  NON-CUBE.  CONSIDER  THE 
SEQUENCES  IN  THE  ORDER  OF  INCREASING  FIRST  TERMS.  CHOOSE  THE 
FIRST  G  SEQUENCES  IN  THIS  ORDER,  IF  THERE  ARE  MORE  THAN  6 
SEQUENCES.  OTHERWISE.  CHOOSE  ALL  THE  SEQUENCES.  POOL  THE 
ELEMENTS  OF  ALL  THE  CHOSEN  SEQUENCES  ANO  SORT  THEM  IN  NUMERICAL 


L 


17 


ORDER.  PLACE  THIS  ORDERED  SEQUENCE  OF  NUMBERS  AT  THE  BEGINNING  OF 
L,  IN  INCREASING  NUMERICAL  ORDER.  PRINT  OUT  L. 


DEFINE  AN  ARRAY  L (10. 10)  ANO  READ  NUflBERS  INTO  THIS  ARRAY  FROM  THE 
FILE  F0R31.GEN.  CHOOSE  A  SUITABLE  FORMAT  FOR  THE  READ  STATEMENT 
BY  FIRST  TYPING  OUT  FCR01.GEN  ANO  EXAMINING  IT.  EXAMINE  ALL  THE 
ROUS  OF  L  AND  IN  EACH  ROW,  IDENTIFY  ALL  SEQUENCES  OF  3  OR  MORE 
NUMBERS  HAVING  ELEMENTS  IN  ThE  FOLLOWING  ORDER  -  A  NUMBER 
DIVISIBLE  BY  7.  AN  ODD  NUMBER,  A  NEGATIVE  NUMBER.  SUCH  A  SEQUENCE 
MAY  START  UiTH  AN.  ONE  OF  THE  SPECIFIED  TYPES  OF  ELEMENTS.  POOL 
THE  ELEMENTS  OF  ALL  THESE  SEQUENCES  AND  SORT  THEM  IN  NUMERICAL 
ORDER.  PLACE  THE  1J  LARGEST  ELEMENTS  ALONG  THE  FIRST  COLUMN  IN 
NUMERICALLY  DECREASING  ORDER  FROM  TOP  TO  BOTTOM.  PRINT  OUT  THIS 
COLUMN. 


DEFINE  AN  ARRAY  L (600)  ANO  FILL  IT  UP  UITH  RANDOM  NUMBERS  IN  THE 
RANGE  FROM  -1000030  TO  1000000.  EXAMINE  THE  ELEMENTS  OF  L  AND 
IDENTIFY  ALL  SEQUENCES  OF  7  OR  MORE  NUMBERS  HAVING  ELEMENTS  EACH 
ONE  OF  WHICH  IS  A  MULTIPLE  OF  i  TS  PREDECESSOR.  POOL  THE  ELEMENTS 
OF  ALL  THESE  SEQUENCES  AND  SORT  THEM  IN  NUMERICAL  ORDER.  PLACE 
THIS  ORDERED  nf*OUENCE  CF  NUMBERS  AT  THE  BEGINNING  OF  L,  IN 
DECREASING  NUMF  AL  ORDER.  PLACE  A  ZERO  IN  ALL  THE  LOCATION'S  OF 
L  UHICH  DO  NnI  CONTAIN  ONE  OF  THE  FINALLY  SELECTED  NUMBERS.  PRINT 
OUT  L. 


F  i ci .  A  Examp  I  es  p_f  ’  Arrau  Order  i na  Problems’ 


PROBLEM 


■SPECIFIED  INPUT.  RANGE  OF  NUMBERS  DECIOEO  HERE 

SPECIFIED  DATA-STRUTTitRE.  ONE  OR  TUO  DIMENSIONAL  ARRAY 

r  SOURCE  OF  OAT A 

SUB-STRUCTURES  OF  GIVEN 

DATA -STRUCTURE  (EG.  ROUS/COLUMNS) 


1ST  MANIPULATION 


OBJECT  TO  BE  SEARCHEO  FOR 
SPECIFIED  SETS 
SPECIFIED  SEQUENCES  OR 
SUBSTRUCTURES  CONTAINING 
SPECIFIED  OBJECTS 


—  ‘ORGANIZATION  OF  OBJECTS  DETECTED 
POOL  OR/ANO  ORDER  SELECTED  OBJECTS 

- RELOCATION  OF  REORGANIZED  OBJECTS 

SPECIFY  DESTINATION 
(SUCH  AS  THF  TOP  3  ROUS, 

MIDDLE  HOW.  LAST  COLUMN) 

■FURTHER  MANIPULATIONS 


OUTPUT  AS  SPEC  I F I EO 


CONSIDER  THE  LIST  L19  AND  THE  UNNAMED  LIST  ON  THE  STACK.  SELECT 
THE  SET  OF  ALL  EVEN  NUMBERS  UH1CH  ARE  PRESENT  IN  LIS  BUT  NOT  IN 
THE  UNNAflEO  LIST.  FIND  THE  SUM  OF  ALL  THE  SELECTED  NUMBERS  AFTER 
ELIMINATING  MULTIPLE  OCCURRENCES. 

YOU  ARE  GIVEN  THE  LI  ST -STRUCTURES  L13  L14  L15  AND  L1G  AND  4 
UNNAMED  LIST-STRUCTURES  ON  THE  STACK.  THEY  MAY  CONTAIN  NUMBERS  AT 
ANY  AND  ALL  LEVELS.  SELECT  THE  SET  OF  ALL  PRIME  NUMBERS  WHICH  ARE 
DRESENT  IN  LIB  .  IN  L15  .  IN  L14  .  AS  UELL  AS  IN  L13  BUT  NOT  IN 
EACH  UNNAMED  LIST-STRUCTURE.  '  I  NO  THE  SUM  OF  ALL  THE  SELECTED 
NUMBERS  AFTER  ELIMINATING  MUL  I PLE  OCCURRENCES . 

CONSIDER  THE  3  UNNAMED  LIST-STRUCTURES  ON  THE  STACK  AND  THE 
LIST-STRUCTURES  LiS  L13  AND  L20.  THEY  MAY  CONTAIN  NUMBERS  AT  ANY 
AND  ALL  LEVELS.  SELCCT  THE  SET  OF  ALL  NEGATIVE  NUMBERS  WHICH  ARE 
PRESENT  IN  L20  BUT  NOT  IN  L19.  MERGE  THESE  WITH  THE  NUMBERS  WHICH 
ARE  GREATER  THAN  100  FROM  THE  TOPMOST  LIST-STRUCTURE  ON  THE  STACK. 
PLACE  ALL  THE  SELECTED  NUMBERS  ON  L18  AFTER  EMPTYING  IT  FIRST. 
SORT  THE  CONTENTS  OF  L18  IN  DESCENDING  NUMERICAL  ORDER.  FiND  OUT 
ALL  SFQUENCES  OF  PRIME  NUMBERS  IN  IT  AND  PLACE  THEM  ON  THE  2  OTHER 
UNNAMED  LIST-STRUCTURES. 


F i g.6  A  Set  of  Simple  Li st-Process ing  Problems 
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Some  of  the  terminal  routines  make  random  choices,  uhi le  others 
access  the  structures  already  assembled  and  choose  compatible 
elements  to  be  used  in  the  sub-structures  being  constructed 
Access  to  a  representation  of  knowledge  of  the  task  area  having 
the  form  described  on  page  4  also  occurs  at  this  level  as  a  part 
of  the  relevance  and  compatibility  computation. 

It  is  useful  at  this  stage  to  evaluate  the  utility  of  the 
selective  mechanisms  discussed  above.  Very  rough  estimates  can  be 
made  of  the  total  number  of  structures  tnat  are  in  oome  sense 
possible,  and  the  fraction  of  these  structures  that  are 
meaningful.  The  number  of  branchings  that  occur  in  the  course  of 
generating  a  problem  of  the  type  shown  in  rig. 4  Is  of  the  order  of 
15.  corresponding  to  the  branchings  of  Figure  5, 

Uhi le  the  choices  that  create  structural  differences  between 
problems  usually  involve  only  two  or  three  alternatives,  e.g.  the 
choice  between  one  dimensional  arrays  and  two  dimensional  arrays, 
numerical  cho:ces  ir.volve  large  numbers  of  alternatives.  Most  of 
the  numerical  choices  are  equivalent,  but  selecting  a  number  in 
the  urong  range  could  create  absurd  probiems,  e.g.,  comparing 
numbers  in  the  range  from  0  to  100  with  the  threshold  of  2508. 
Estimating  that  there  are  2  or  3  significant  alternatives  at  most 
branches,  the  total  number  of  choice  sequences  that  are  available 
over  15  steps  is  in  the  range  from  10T4  to  18T7  .  Typically, 
five  or  six  of  these  decisions  are  contingent  upon  others,  so  that 
the  amount  of  selection  exercised  by  the  program  in  producing  a 
coherent  problem  (out  of  what  would  be  mostly  nonsense  problems  if 
free  choices  were  made  at  each  point)  is  roughly  1  in  503. 
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3.2  Assemb I u  of  Sentences 

What  is  the  nature  of  the  stru'ture  assembled  by  a  problem 
generator  in  relation  to  a  cnmprehendable  external  description  of 
the  problem?  One  possibility  is  that  the  structures  assembled  by 
the  generator  be  directly  the  deep-structures  of  sentences  which 
would  constitute  a  readable  description  of  the  problem.  In  this 
case  all  the  knowledge  required  to  translate  these  structures  into 
readable  descriptions  can  be  localized  in  a  ‘sentence 
synthesizer’.  Such  a  synthesizer  uould  remove  the  burden  of 
syntactic  considerations  from  the  problem-generator.  The  input 
requirements  of  this  synthesizer  would  define  a  range  of  possible 
formats  for  the  data-structures  to  be  created  by  the  generator, 
ule  have  implemented  such  a  sentence  synthesizer  (see  Simmons  and 
Slocum,  1372  for  a  detailed  treatment  of  sentence  synthesis).  The 
problems  in  Figures  G,  S,  12  and  13  were  all  generated  by  programs 
which  first  created  we  I  I -def ; ned  deep-structures  for  their 
sentences.  These  were  then  rendered  into  readable  form  by  the 
sentence  synthesizer.  A  brief  discussion  of  the  nature  of  the 
sentence  structures  used  is  given  here.  Some  information  on 
programming  considerations  and  on  the  sentence  synthesizer  may  be 
found  in  Appendices  A  and  B. 

The  deep-structures  are  in  the  form  of  labelled  directed 
graph-structures.  Thr>  nodes  are  unnamed  lists  representing 
specific  occurences  (or  ’tokens’)  of  concepts  symbolized  by 
word-roots,  numbers  or  symbols.  The  edges  are  associations 
implemented  by  using  association  lists  and  a  hash-coding  scheme. 
While  the  list  itself  represents  the  token,  its  content,  genera.  iy 
a  word-root  (referring  to  a  specific  word-sense,  in  the  case  of 
ambiguous  words),  represents  the  ’type’.  There  may  be  several 
lists  with  the  same  content  to  represent  different  tokens  of  the 
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same  type.  Structures  are  formed  by  creating  labelled  (and 
directed)  associations  between  nodes  to  represent  deep-case 
relations  between  the  entities  referred  to  in  a  sentence.  Some  of 
the  functions  performed  by  these  relations  are  i I 'ustrated  by  the 
examples  in  Fig. 7.  the  two  appendices  describe  briefly  the 
construction  and  interpretation  of  such  sentence-structures. 

White  token-to-token  associations  create  sentence  structures, 
other  associations  involving  types  embody  the  sentence 
synthesizer’s  knowledge  of  the  language.  For  example,  such 
associa4  link  words  to  their  syntactic  categories  and  to  their 
excep'  plural  forms  when  these  exist.  They  also  link 
syntac  categories  with  appropriate  synthesis  routines  which 
recognize  basic  sub-structures  of  sentences  and  have  the 
capabilities  to  generate  appropriate  word-groups  for  them.  Uhile 
token-to-token  associations  are  dynamic  in  the  sense  of  being 
created  and  destroyed  as  sentences  are  generated  and  printed  out, 
the  syntactic  and  semantic  associations  are  permanent.  However, 
the  same  association  median  ism  in  the  programming  facility 
provides  the  three  types  of  associations  discussed  so  far: 
associations  embodying  information  on  the  task  area  as  described 
on  page  4,  the  token-to-token  links  which  are  used  to  create 
sentence  structures  and  the  syntactic  and  semantic  associations 
embodying  the  sentence  synthesizer’s  knowledge  of  English.  As  may 
oe  expected,  there  is  some  overlap  between  what  should  belong  to 
the  first  type  and  what  should  belong  to  the  third  type. 


Fia. 7  Deen  Structure  cl  2  One-Sentence  Pr gb I t?n 
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!i2  Contro I  of  the  Generat i ve  Process 

In  discussing  programs  having  the  control  structure  described 
in  Section  3.1  it  is  useful  to  talk  in  terms  of  a  search  of  the 
’design  space’  which  is  a  special  case  of  the  ’action  space’, 
using  the  search  representation  for  problem-solving  (Newell  and 
Simon,  1972).  The  series  of  design  decisions  that  leads  to  the 
assembly  of  a  suitable  object  is  a  special  case  of  the 
’  so  I  ut  i  on-ac  t  i  on-set;  fence ' . 

Complementing  this  description  is  the  set  representation  fur 
problem-generation  which  deals  with  the  target  space  of  all 
problems  that  can  be  produced  by  the  generator.  Starting  at  the 
highest  level,  the  program  selects  progressively  smaller 
subspaces,  till  it  identifies  a  single  point  in  it  which 
represents  the  problem  generated  at  this  attempt.  The  design  of 
the  program  has  to  start  with  the  visualization  of  a  rich  enough 
space.  The  control  structure  has  to  ensure  that  a  I  !  the 
visualized  space  is  accessible  by  a  suitable  sequence  of  choices. 

An  important  question  is  concerned  with  the  control  of  this 
access  path.  How  can  one  influence  a  series  of  choices  selecting 
one  possible  access  path  out  of  a  large  number  all  of  which  lead 
to  meaningful  problems? 

A  powerful  and  yet  simple  control  mechanism  that  provides  a 
certain  degree  of  direction  in  the  generative  process  depends  upon 
the  use  of  a  quasi-formal  language  to  specify  desired  sets  of 
problems.  For  example,  problems  of  the  type  in  Fig. 8  are  easily 
produced  by  appropriate  interpretation  of  brief  high-level 
descriptions,  e.y.: 
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(CREATE  A  LIST) 

(SCAN  A  CIRCULAR  LIST-STRUCTURE) 

(SUBSTITUTE  COMPONENTS  IN  A  SYMMETRIC  LIST-STRUCTURE) 
(FIND  PROPERTY  OF  GIVEN  LIST-STRUCTURE) 

(TEST  IF  A  OATA-STRUCTURE  IS  RECURSIVE) 

(DELETE  COMPONENTS  FROM  A  SYMMETRIC  LIST-STRUCTURE) 

A  sentence  in  such  a  language  predetermines  a  specific  set  of 
choices  to  be  made  by  the  generator,  leaving  other  choices  to  be 
made  freely  Naturally,  the  structure  of  the  sentence, 
interpreting  mechanism  will  determine  the  effectiveness  of  any 
such  scheme.  The  interpreting  mechanism  has  to  analyse  the 
sentences  and  determine  uhich  steps  of  the  generative  procesc  are 
being  forced,  and  in  uhich  direction. 

An  appealing  form  for  such  an  interpreter  is  a  semantic 
o-  Tpr  uhich  find?  out  the  deep-case  relationships  betueen  the 
entities  referred  to  by  a  sentence,  sharing  syntactic  and  semantic 
information  used  by  the  sentence  synthesizer. 
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CREATE  A  LIST  CONTAINING  THE  FIRST  42  PRIME  NUMBERS 
> 

COUNT  OCCURRENCES  OF  SEQUENCES  OF  NEGATIVE  NUMBERS  IN  THE  CIRCULAR 

LIST-STRUCTURE  GIVEN 
> 

SUBSTITUTE  ALL  OCCURRENCES  OF  THE  COMPONENT-LIST  C2  IN  THE  SYMMETRIC 

LIST-STRUCTURE  GIVEN  BY  THE  TERMJNA'  -NODE  GIVEN 
> 

URITF  A  PROGRAM  TO  CONCATENATE  THE  LISTS  LIST-L1  AND  LST-L2 
> 

bUBSTITUTE  THE  LAST-BUT-ONE  OCCURRENCE  OF  THE  TERMINAL-NODE  T3  IN 

THE  SYMMETRIC  LIST-STRMCTURE  L3  BY  THE  COMPONENT-LIST  GIVEN 
> 

FIND  THE  LENGTH  OF  THE  LIST-STRUCTURE  GIVEN 
> 

FIND  IF  THE  CIRCULAR  LIST-STRUCTURE  LI  IS  RECURSIVE 
> 

URITE  A  PROGRAM  TO  CONCATENATE  THE  LISTS  LIST-L1  AND  LIST-L2 
> 

DELETE  ALL  OCCURRENCES  OF  TERM INAL -NODES  OF  THE  TYPE  I  IN  THE 

SYMMETRIC  LIST-STRUCTURE  GIVEN 

-1  fl ♦  &  A  Set.  cf  Prob I ems  Generated  bu  Interpret  i nn 

Brief  Problem  Speci f i cat  ions 
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The  high-level  descriptions  of  problems  indicate  to  objects 
and  processes  which  are  to  be  referred  to  in  the  problem  to  be 
generated.  Reference  is  made  to  them  directly  or,  sometimes, 
indirectly  by  mentioning  sets  of  entities  from  which  the  generator 
may  choose  suitable  members.  Fig. 3  shows  some  of  the  objects  and 
processes  referred  to  by  the  problems  being  discussed  and  their 
organization  into  progressively  larger  sets.  The  interpreter  has 
to  recognize  case  relationships  to  find  the  function  each  entity 
is  to  perform  in  the  problem  and  use  this  information  to  control 
the  generative  process. 

The  simple  interpreter  which  produced  the  problems  of  Fig. 8 
from  high-level  descriptions  was  implemented  by  defining  a  ’search 
routine’  for  each  word  category,  the  categorization  being 
basically  syntactic.  Interpretation  is  performed  by  executing  the 
search  routine  associated  with  each  word,  allowing  it  to  find 
other  words  in  the  sentence  which  are  related  to  i*.  This  process 
results  in  the  construction  of  parts  of  deep-structures  of 
ootential  sentences  to  describe  entities  being  referred  to  and 
their  relationships  to  each  o‘her.  The  associative  schemes 
referred  to  on  page  24  are  useful  here  too.  After  this  parsing, 
the  interpreter  triggers  off  the  problem  generator  giving  it  the 
references  to  the  entities  it  must  use  in  \  ‘  ,i  operation.  The 
generator  processes  any  indirect  references  to  obtain  suitable 
entities  which  are  to  be  used  in  problem  generation  and  completes 
i ts  task. 
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objects 


I i st/ I i st-structure 

circular  I i st/ i i st-structure 
symmetr ical  I i st/ I i st-structure 


processes 


.  create 


examme^ 


..find  proper  t  ies 

.scan  for  specified 
content;. 


.count  them 


print  them 


I _ mod  i  fy 


i  ntroduce 


L.  delete 


concatenate 

subst i tute 


sub-structures 


component  objects. 


terminal  nodes 


,  numbers 


symbo  I  s 


F  i  a .  9  A I ternat i ves  for  the  Choice  of  Components  for  ih§. 

Problems  in  Fig. 8 
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Uhile  explicitly  specified  choices  are  easily  forced  upon  <.iie 
problem  generator,  there  are  several  possibilities  for  handling 
the  choices  that  are  not  forced.  One  such  possibility  is  that 
information  about.  the  generation  of  sub-problems  can  be 
accumulated  in  the  course  of  generating  a  series  of  problems. 
This  information  can  then  be  used  to  ensure  that  sub-problems  are 
well  distributed  in  a  generating  run.  For  instance,  from  the 
knowledge  that  a  particular  series  of  tasks  have  been  successfully 
carried  out  by  a  student,  a  set  of  sub-problems  that  he  has 
carried  out  can  often  be  identified.  It  can  then  be  ensured  that 
later  problems  do  not  incorporate  these  sub-problems  already 
familiar  to  the  student,  uniess  there  are  special  reasons.  There 
is  also  the  possibility  that  hierarchic  relationships  between 
tasks  can  be  exploited  to  produce  sequences  of  problems  of  graded 
difficulty.  Fig. 10  shows  recognizable  sub-tasks  which  are 
incorporated  in  the  problems  of  Fig. 4,  organized  to  exhibit 
hierarchic  dependencies.  The  sub-  tasks  appearing  in  underlined 
script  are  complete  in  the  sense  of  being  usable  as  self-contained 
exercises.  App  opriate  organization  of  the  generator  should  allow 
such  exercises  to  be  produced  when  needed,  without  use  of  any 
additional  machinery. 
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SOLVING  ARRAY  PROCESSING  PROBLEMS 
OF  THE  TYPE  ILLUSTRATED  IN  FIG. A 


REARRANGING  AN  ARRAY  BY 
MOVING  SEQUENCES  IN  U 
U1THOUT  LOSING  ITS  CONTENTS 


# 

ACCUMULATING  SETS  OF 
NUMBERS  THAT  HAVE 
BEEN  SELECTEO  FOR 
LATER  USE 

SCANNING  AN  ARRAY. 
SELECTING  SPECIFIED 
ELEMENTS 

I 

A 


STORING  REFERENCES 
10  SEQUENCES  IN  AN 
ARRAY  FOR  LATER  USE 

l 


ACCESSING  ELEMENTS  OF 
THE  ARRAY  AND  TESTING 
THEM  BY  COMPUTING 
NUMERICAL  PREDICATES 


SCANNING  AN  ARRAY 
SEARCHING  FOR 
SEQUENCES  QF 
SPECIFIED 
STflUCTURE 


REARRANGING  CONTENTS 
QF  AN  ARRAY  BY 
EXCHANGE  AND 
SUBSTITUTION 


CODING  INPUT  FROM 

OiSCiliE 

# . 

QOQlflG  OUTPUT  IQ 

P.15QJE1LE 


FILLING  ARRAY 
141  TH  RANDOM  NUMBERS 


F  i  q. 10  Sub-Tasks  Encountered  in  the  Arrau  Process i na  Prob I ems  of  F i a ■  4 
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!L.  SENERAJLIZAILQN  QE  A  PROTOTYPICAL  PROBLEM 

1  it  possible  to  start  from  a  well-known  problem  and  write  a 
generator  for  producing  useful  generalizations  of  the  original 
problem?  This  section  describes  experiments  with  this  technique  of 
generalization,  applying  it  to  a  problem  fi-st  discussed  by 
Hoare(1964)  and  then  by  Djikstra  (1971).  The  problem  is; 

Rearrange  the  elements  of  the  array  A [1 : N]  such  that 
for  a  given  value  of  f  ( 1 < f <N) 

A[kl<Alfj  if  1  <k< f  and 
A  If ] <A  (kl  i f  f <k<N  . 

The  roqui'-ed  arrangement  is  said  to  ’split*  the  array  around  f. 

As  a  programming  exercise  this  task  has  manifold  appeal.  It 
confronts  the  student  with  the  problem  of  planning  manipulations 
which  interact  heavily.  If  ue  eliminate  the  brute  force  solution 
of  sorting  the  array,  perhaps  by  adding  an  economy  clause  to  the 
problem  statement,  the  solution  is  no  longer  obvious.  Djikstra 
describes  an  elegant  solution  based  on  recursion. 

Ue  look  at  it  from  another  point  of  view,  attempting  to 
generalize  it  into  a  class  of  array-order  iny  problems  which  test 
the  student’s  understanding  of  the  concepts  relevant  to  sorting. 
Ue  do  not  specifically  concern  ourselves  with  the  existence  or 
non-existence  of  an  eleg3nt  recursive  solution  to  the  problems 
generated. 

Fig. 11  lists  some  of  the  skills  needed  to  program  basic 
sorting,  in  an  order  intended  to  indicate  the  dependence  of  some 
of  these  skills  on  others,  the  more  basic  ones  being  generally 
lower  in  the  list. 
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REARRANGING  ARRAY  AS  SPECIFIED 

MANIPULATING  ELEMENTS  OF  AN  ARRAY  L0CALL7 
(SUBSTITUTIONS  AND  INTERCHANGES) 

PERFORMING  ARITHMETIC  ON  ELEMENTS  OF  ARRAY 
(COMPUTING  FUNCTIONS  AND  PREDICATES 
DEFINED  OVER  ELEMENTS  OF  THE  ARRAY) 

SCANNING  ARRAY  FOR  SPECIFIED  ELEMENTS 

ACCESSING  SPECIFIED  ELEMENTS  OF  ARRAY 
BY  GENERATING  DESIRED  SEQUENCE  OF  SUBSCRIPTS 

Fiq.il  E I ements  of  the  Sor t i nq  Exerc i se 

In  terms  of  these  operations,  sorting  turns  out  to  be  a 
rearrangement  of  u  given  array  A(1:N]  such  that  Atj]  >  A(i)  if 
j  >  i  . 

A  simple  generalization  of  the  Hoars  problem  which  still  has 
the  listed  concepts  as  essentials  is: 

Rearrange  elements  of  the  array  A(i:NI  such  that 
if  R 1  ( < i > , c j > )  then  R2 (A< i >, A< j>) 
where  Ri  and  R2  stand  for  relations  and  <i>  and  <)>  are 
generator  variables  which  are  to  be  substituted  by  a 
variable  such  as  i.j.k  or,  when  suitable,  by  a  numerical 
constant. 
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A  related  set  of  problems  is  of  the  form: 

Rearrange  elements  of  array  A tl : N J  such  that 
i  f  PI  Ui>)  then  P2(A(<i>] ) 
where  PI  and  P2  are  predicates. 

Fig. 12  shows  some  of  the  problems  generated  by  a  program 
based  on  the  two  generalizations  indicated  above.  Comparison  of 
these  problems  with  those  presented  in  Section  3  shows  that  these 
problems  have  an  element  of  programming  difficulty  not  generally 
present  in  the  earlier  problems.  The  programmer  now  has  to  tackle 
the  question  of  space  needed  by  the  contents  of  the  array.  Moving 
the  content  of  a  cell  requires  that  space  be  found  for  it 
elsewhere.  Creatir.g  space  could  involve  disturbing  elements  which 
have  been  properly  placed  already  and  considerable  interaction 
between  desirable  changes  is  caused  thereby.  This  is  a 
significant  aspect  of  the  original  problem  that  is  preserved. 
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ARRANGE  THE  MUMBERS  IN  THE  ARRAY  A (1,40]  SUCH-THAT  THE  ELEMENT  IN 
CELL  AU]  IS  A  SQUARE  IF  l  IS  A  CUBE. 

> 

ARRANGE  THE  NUMBERS  IN  THE  ARRAY  A 11, 251  SUCH-THAT  THE  ELEMENT  All] 
IS  COMMENSURATE  WITH  THE  ELEMENT  A  [111  UHEN  .  IS  LESS  THAN  11. 

> 

AhRANGE  THE  NUMBERS  IN  THE  ARRAY  A II. 301  SUCH-THAT  THE  ELEMENT  All! 
IS  LESS  THAN  THE  ELEMENT  A 1131  UHEN  I  IS  DIVISIBLE  BY  13. 

> 

ARRANGE  THE  NUMBERS  IN  THE  ARRAY  All.  -293  SUCH-THAT  THE  ELEMENT  IN 
ANY  AN  EVEN-NUMBERED  CELL  IS  A  SQUARE. 

> 

ARRANGE  THE  NUMBERS  IN  THE  ARRAY  A  11,301  SUCH-THAT  THE  ELEMENT  All] 
IS  LESS  THAN  THE  ELEMENT  A 17]  UHEN  I  IS  COMMENS'  RATr  U1TH  7. 

> 

ARRANGE  THE  NUMBERS  IN  THE  ARRAY  A [1,40]  SUCH-THAT  THE  ELEMENT  All] 
IS  COMMENSURATE  UI TH  THE  ELEMENT  AIJ]  UHEN  I  IS  COMMENSURATE  WITH  J. 


Fig. 12 


Arrau  Order i ng  Problems 
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A  sequence  of  several  problems  derived  from  the  same 
prototype  would  be  unsuitable  for  presentation  to  a  student,  if 
they  show  no  more  variation  than  the  set  of  problems  in  Fig. 12.  A 
library  of  prototypes  could  provide  some  coverage  and  variety.  On 
the  other  hand,  focus  on  selected  concepts  would  be  useful,  even 
at  the  expense  of  variety,  uhen  producing  problems  for  certain 
purposes:  for  introducing  important  concepts,  for  testing,  for 
remediation  or  for  review.  If  the  student  has  a  convenient 
addressing  facility,  he  would  often  find  it  useful  to  ask  for  a 
few  similar  problems  involving  e  specific  set  of  concepts  to 
confirm  his  own  understanding  of  these,  before  he  moves  on  to 
other  concepts. 
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^  reasoning  about  actions 

The  basic  characteristic  of  a  problem  situation  is  that  the 
scheme  of  actions  leading  to  a  solution  is  not  obvious,  while  the 
elements  out  of  which  such  a  scheme  could  be  constructed  ate  more 
or  less  available.  It  is  generally  important  in  problem 
generation  to  estimate  and  control  the  complexity  of  the 
action-schemes  which  might  lead  to  sr  utions,  ensuring  that  the 
problem  presents  a  desired  level  of  difficulty  to  the  student.  To 
some  extent,  the  generator  may  also  predetermine  the  content  and 
structure  of  fruitful  action-schemes. 

One  device  for  the  estimation  of  complexity  of  a  problem 
involves  the  visualization  of  possible  problem  spaces  that  might 
be  searched  for  a  solution.  In  the  case  of  programming  exercises, 
the  cost  of  searching  the  most  easily  intuited  space  of  solutions 
can  often  be  estimated  easily  by  the  generator.  Ensuring  this 
cost  is  too  high  forces  a  step  of  refinement  on  the  student, 
making  him  look  for  clues  for  augmenting  the  problem-space  or,  in 
some  cases,  compelling  him  to  restructure  the  program,  for  example 
by  providing  for  storage  of  computed  values  to  reduce 
computational  cost  or  by  substituting  iteration  for  recursion. 
Uhen  dealing  with  well-structured  problems,  a  lower  bound  for  the 
complexity  of  the  program  needed  to  solve  a  given  problem  may  be 
estimated  by  identifying  those  features  of  the  program  which  have 
been  rendered  essential.  A  generator  could  use  such  estimates  to 
arrive  at  problems  presenting  desired  levels  of  difficulty. 

A  second  device  that  could  be  employed  is  the  programming  of 
limited  deduction  about  the  problem  situation.  The  problsms 
appearing  in  underlined  script  in  Fig. 13  provide  good  examples  of 
situations  where  the  scheme  of  actions  needed  to  perform  the  given 
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task  is  of  controlled  complexity  (1).  Each  of  these  problems 
involves  an  action  or  a  set  of  actions  that  is  to  be  performed  in 
order  to  achieve  a  desired  change  or  to  achieve  a  desired  state  in 
the  given  situation.  The  *ask  is  meaningful  only  if  the  action 
specified  is  relevant  to  the  state  change  desired  and  if  the 

performance  of  this  action  presents  a  desired  level  of  difficulty. 
In  order  to  take  these  requirements  into  account  the  problem 

generator  has  to  have  information  relating  possible  actions  to 

their  effects,  it  also  needs  estimates  of  difficulty  in  carrying 
out  possible  actions  (i.e.  'applying  operators’,  to  use  the 
wording  in  Newell  and  Simon,  1972).  The  use  of  a  table  of 

connections  (ibid)  is  a  possibility.  Deducing  the  information  by 
following  a  suitable  line  of  reasoning  f..ay  be  necessary  in  those 
cases  wnere  a  table  of  connections  becomes  unwieldy. 

The  problems  under  discussion  deal  only  with  a  few  possible 
types  of  simple  manipulations  that  can  be  performed  on  graphs, 
i.e.  addition  or  deletion  of  vertices  and  edges  and  the 
repositioning  of  edges.  Each  type  of  manipulation  is  generally 
capable  of  multiple  application  to  a  graph. 


(1)  The  other  problems  shown  in  this  figure  are  not  discussed 
here.  They  were  produced  by  generators  having  the  structure 
discussed  in  Section  3  and  share  association  networks  with  the 
problems  being  discussed  ir>  this  section. 
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>  DELETE  LF.5S  THAN  2  EDGES  FROM  I*£  GRAPH  Q  IQ  JNCBEASE  IIS  BADjUS. 

IQ  2j_ 

>  ASSIGN  NUMBERS  IN  THE  RANGE  FROM  1  TO  10  TO  THE  EDGFS  OF  THE  GIVEN 
GRAPH  SUCH  THAT  THE  NUMBER  ASSIGNEO  TO  AN  EOGE  IS  COMMENSURATE  WITH 
THE  TOTAL  NUMBER  OF  THE  EDGES  ADJACENT  TO  IT.  ENSURE  THAT  THE  SUM  OF 
THE  ASSIGNED  NUMBERS  IS  A  MAXIMUM. 

>  FIND  BRIDGES  WITH  A  LENGTH  OF  4  IN  THE  GIVEN  GRAPH. 

>  ADQ  LESS  THAN  &  EDGES  TO  THE  GRAPH  G  SUCH  IHAJ  U  IS  NQ  IQNGEfi  4 
PLANAR  GRAPH. 

>  FIND  IF  THERE  ARE  CYCLES  IN  THE  GRAPH  G. 

>  ADQ  LESS  THAN  3  N£U  VERTICES  U1TH  NECESSARY  I  NCI  PENT  E DOES.  IQ  IHE. 
GRAPH  G  SUCH  IHAT  THE  VERTICES  V7  V2  Via  V6  AND  VI  ARE  NO  LONGER 
THE  CENTRAL  /ERT1CES. 

:•  FIND  CYCLES  IN  THE  GRAPH  G. 

>  ASSIGN  A  COLOR  NAME  TO  EACH  EDGE  SUCH  THAT  THE  COLOR  OF  ANY  EOGE 
IS  DIFFERENT  FROM  THE  COLOR  OF  ALL  ITS  ADJACENT  EDGES. 

>  ASSIGN  NUMBERS.  NOT  NECESSARILY  UNIQUE.  IN  THE  RANGE  FROM  1  TO  IS 
TO  THE  EDGES  OF  THE  GRAPH  G  SUCH  THAT  THE  NUMBER  ASSIGNED  TO  AN  EDGE 
IS  MULTIPLE  THE  NUMBER  OF  THE  CYCLES  IT  IS  A  MEMBER  OF. 

>  GENERATE  BRIDGES  WITH  A  LENGTH  OF  4  FROM  THE  GRAPH  G. 

>  FIND  IF  THE  GIVEN  GRAPH  IS  A  BLOCK. 

>  REPOSITION  3  OR  MORE  EDGES  ]N  M  GRAPH  £  SUCH  THAT  IH£  EQCEMTRJ.C]  TY 
OF  V9  REMAINS  UNCHANGED. 


QaJl 


Graph  Manipulat ion  Prob I emg 
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Some  of  the  problems  require  that  the  graph  be  modified  to  attain 
a  given  Property  by  performing  only  a  limited  number  of  a 
specified  type  of  manipu i at i on.  Other  problems  require  that  a 
given  number  of  manipulations  of  a  specified  type  be  performed 
without  alter.ng  the  graph  in  a  specified  uay.  Since  these  are 
programming  problems,  *he  real  task  is  to  write  a  prog,  am  which 
performs  the  required  manipulations  on  ^iy  graph  given  as  its 
i npu  t . 


The  problems  are  interesting  because  performance  of  the  task 
under  the  given  constraints  becomes  a  critical  matter,  requiring 
considerable  analysis  of  the  problem  situation  and  careful 
planning  of  the  solution.  A  special  difficulty  is  that  the 
solution  should  be  applicable  to  an  indefinitely  large  set  of 
graphs  any  of  which  may  be  given  as  input  to  the  program  that  is 
to  be  written.  These  features,  and  the  presence  of  a  body  of 
theory  on  graphs,  make  these  problems  good  exercises  in 
non-numer ica I  programming. 

The  generator  which  produced  the  examples  in  i"ig,13  performs 
limited  deduction  to  arrive  at  the  relation  between  the  five 
allowed  manipulations  on  graphs  to  a  dozen  or  so  state  changes 
that  could  occur  in  the  graphs  as  a  result.  In  the  case  of  the 
problems  in  Fig. 13,  the  deduction  is  based  on  the  heuristic  use  of 
knowledge  about  gross  changes  that  occur  in  the  graph  as  a  result 
of  simple  manipulations.  Tuo  gross  indicators  are  the  change  in 
the  number  of  Daths  between  an  arbi trary  pair  of  points  as  a 
result  of  a  manipulation  aid  the  change  in  distance  between  an 
arbitrary  pair  of  points.  table  1  relates  manipulations  to  their 
effects  in  terms  of  these  gross  indicators.  Table  2  relates  these 
indicators  to  several  state  charts  that  can  be  produced  in 
graphs.  In  these  tables,  an  entry  of  a  *1’  shows  a  tendency  for 
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the  cause  to  increase  the  quantity  it  is  related  to;  and  an  entry 
of  a  ’ -1*  shows  the  opprsite  tendency;  and  an  entry  of  ’0* 
indicates  that  a  predictable  link  is  not  present. 


Operation 

Delete  Vertices  with  edges 
Delete  Edges 
Add  Edges 

Add  Vertices  and  edges 
Reposition  Edges 


Change  j_n  Change  in 

Path  Length  Number  of  Paths 


+1 

-i 

+1 

-1 

-1 

+1 

0 

+1 

0 

0 

Operators  and  Their  Effect  on  Graph  Parameters 


Tab : e  l 
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Siaifi  Increase  j_n 

Degsr.iPtPca  Eaiii  Length 


Increase  in  Number. 

01  Earns 


Radius  +1 

Diameter  +1 

Eccentricity  +1 

Connectivity  0 

Number  of 

Components  0 

Planarity  0 

Possibi I i ty  of 

Hamiltonian  Paths  0 

1  dent  i ty  of 

Center  0 

Distance  Between 
Two  Nodes  +1 

Possibi I i ty  of 

Euler ian  Paths  0 


-1 

-1 

-1 

+1 

-1 

-1 

+1 

0 

-1 

0 


Iafa La  Z  5l3lg  Changes  gnji  Associated  Variations  in 


GnSfih  Parameters 
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To  generate  the  examples  rsierred  to,  the  generator  chooses  a 
permitted  manipulation  and  identifies  a  re'evant  state  change  by 
using  the  tables.  Further,  it  computes  from  these  tables  the 
direction  of  state  change  and  introduces  a  suitable  constraint 
uhich  ensures  that  the  task  being  constructed  is  not  too  easy. 

Extensions  can  easily  be  made  to  the  generator  employing  the 
line  of  reasoning  described  above.  Consider,  for  instance,  the 
problems  uhich  require  that  a  particular  state  change  be  realized 
uithout  altering  a  bpecified  feature  of  the  given  graph.  A  simple 
computation  involving  Table  2  enables  the  generator  to  produce 
pairs  of  state  changes  requiring  opposite  directions  of  change  in 
the  tuo  parameters  involved.  A  reference  to  Table  1  identifies  a 
relevant  mani  pul  at  i'”~  on  <ne  graph.  These  tuo  steps  enable  the 
generator  to  create  a  problem  by  requiring  that  the  specified 
manipulation  be  repeatedly  performed  to  obtain  a  given  3tate 
change  ensuring  that  the  other  state  change  does  not  occur. 
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CQYE S  SLQRiis 

Large  classes  of  problems  can  be  viewed  as  having  an  abstract 
structure.  In  these  problems,  *he  specific  objects  functions  or 
relations  that  appear  are  not  of  much  significance,  and  are 
generally  substituted  by  formal  variables  early  in  the 
problem-solving  process.  The  same  abstract-structure  obtained 
from  many  problems  which  differ  oniy  trivially  describe  the 
essence  of  the  problem  situation.  School  text-books  on 
mathematical  subjects  employ  the  well-known  technique  of  taking 
interesting  problem-structures  and  fleshing  them  out  with 
sufficient  surface  detail  to  generate  exercises.  It  is 
conceivable  that  problems  generated  by  this  technique,  apart  from 
providing  variety  ,  also  focus  attention  on  the  skills  necessary 
to  abstract  the  basic  structure  of  given  problems  (1).  The 
process  of  creating  a  coherent  surface  structure  is  worth  3  brief 
discussion,  even  though  ue  have  not  yet  implemented  a  program  to 
i I  I ustrate  i t . 

At  the  Iojes'.  level  of  sophistication  would  be  schemes  which 
operate  with  i  fixed  number  of  abstract  problem-structures  and  3 
fixed  number  if  surface  structures.  Using  some  indexing  scheme,  a 
compatible  pair  of  structures  would  be  selected  and  the  formal 
varibles  in  the  abstract  structure  would  be  replaced  by  the 
corresponding  terms  of  the  surface  structure.  At  the  other  end  of 
the  spectrum  would  be  an  under standing  system  which  has  stratc  es 
for  finding  or  creating  real-world  situations  corresponding  to  any 
given  abstract  problem-structure. 


(1)  Hayes  and  Simon  (1373)  are  studying  subjects’  behavior  when 
presented  with  differently  structured  external  cover  stories  for 
the  same  abstract  problem  structure  (tower  of  hanoi). 
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Possible  compromises  between  what  is  desirable  emu  «nat  is 
implementable  include  systems  which  exploit  strong  links  between  a 
large  body  of  theory  ot  a  class  of  structures  and  a  related  area 
of  ’real -world'  problems  with  adequate  diversity.  Consider,  for 
instance,  problems  involving  transport  and  communication  networks. 
It  is  obvious  that  many  graph  manipulation  problems  can  be 
paraphrased  as  problems  in  this  area.  Implementing  a  system  which 
can  perform  such  a  paraphrasing  intelligently  involves  several 
problems.  Such  a  system  could  use  a  set  of  associations  linking 
entities  in  the  abstract  theory  with  corresponding  entities  in  the 
application  area.  A  partial  mapping  may  be  possible,  and  it  will 
indicate,  to  some  extent,  those  ascents  of  the  theory  that  are 
relevant  to  the  application  area.  For  example,  the  concept  of  a 
weighted  graph  is  specially  relevant  to  the  application  area  being 
considered  here  because  it  provides  for  the  possibility  of  taking 
into  account  physical  lengths  of  the  edges. 

Unless  such  information  on  relevance  is  used  to  shape  the 
aostract  problem  being  created,  only  a  very  small  fraction  of 
generated  structures  can  be  paraphrased  as  natura I  - 1 ook i ng 
application  area  problems,  Hrwever.  the  complexity  of  detail  that 
has  to  be  attended  to  appears  formidable.  This  is  best 
illustrated  with  examples.  In  the  area  of  transport  and 
communication  networks,  some  of  the  information  that  should  be 
available  to  the  generator,  directly  or  indirectly,  is: 

Road  networks  which  are  non-planar  have  to  be 
constructed  with  the  use  of  underpasses  and 
overpasses  which  are  expens  ve  in  relation  to  the 
cost  of  roads.  This  is  not  the  case  with 

air-routes  and  telecommunication  circuits. 

The  presence  of  cut-vertices  and  small  cut-sets  is 
relate'1  to  the  problems  of  reliability  of  the 
network. 
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Paths  passing  through  every  edge  (Euler ian  paths) 
a<-e  relevant  to  bus  routes  (edges  being  streets), 
uhi  le  they  are  not  of  significance  in  the  case  of 
telephone  networks. 

it  is  possible  to  think  of  railroad  stations  as 
fuelling  or  non-fuelling  stations  (say  for 
generating  problems  involving  dominant  sets  of 
vertices  in  a  graph).  This  is  generally  not  true 
in  the  case  of  airports,  all  of  which  can  provide 
fue  I . 

These  examples  illustrate  the  wealth  of  knowledge  that  a 
human  designer  brings  to  bear  on  the  task  of  creating  cover 
stories.  Programming  a  generator  to  design  cover  stories 
therefore  has  to  face  the  problems  of  representing  the  basis  for 
such  knowledge  and  of  utilizing  it. 


Zj_  conclusions 


While  simple  generative  schemes  do,  in  seme  cases,  produce 
surprisingly  coherent  questions,  it  is  clear  that  the  creation  of 
a  good  problem  is  in  general  a  design  task  k’hich  involves 
considerable  knowledge  of  the  task  area.  Detailed  representations 
of  relevant  knowledge  of  the  task  area  are  as  important  in  problem 
generators  as  they  are  in  the  case  of  any  program  which  performs  a 
cognitively  significant  task.  These  representations  structure  the 
task  area  in  terms  of  objects  and  processes,  assign  attributes  to 
these  entities  and  provide  static  links  as  well  as  computational 
routines  to  implement  functional  and  relational  mappings. 

Given  such  a  representation  of  knowledge,  the  questions  that 
arise  about  a  problem  generator  concern  its  structure  and  the 
strategies  that  it  incorporates.  Are  generators  arbitrary 
programs  which  have  to  be  conceived  ad-hoc,  or  is  there  a  common 
set  of  principles  theu  can  be  based  upon? 

Ue  have  developed  the  view  that  generators  perform  a  design 
task,  assembling  a  complex  object  -  the  problem  -  while  operating 
within  a  variety  of  constraints.  A  top-down  generative 
control -structure  appears  attractive.  The  second  suggestion  was 
that  this  generative  mechanism  be  freed  from  the  burden  of 
syntactic  consic'crat  ions  it  carries  in  the  case  of  grammar- 1  i  ke 
mechanisms.  The  data-structures  it  operates  with  could  be 
deep-structures  of  sentences,  providing  a  rich  representational 
scheme  easily  extended  by  borrowing  from  natural  language.  A 
sentence  synthesizer  which  creates  readable  descriptions  of 
generated  problems  was  described. 
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Problem  generators  for  specific  areas  have  to  be  developed  by 
inventing  appropriate  representations  for  knowledge  about  problems 
in  that  area.  Some  sets  of  problems  have  a  prototype  and  a 
process  of  uhat  might  be  called  semantic  generalization  results  in 
a  set  of  problems  which  are  similar  to  the  original  and  yet 
provide  significant  diversity.  Some  sets  of  problems  can  be 
created  by  performing  limited  deduction  about  operators  and  state 
changes  they  cause  in  the  relevant  problem  space.  These 
strategies  have  been  discussed  and  illustrated  with  examples 
produced  by  programs  based  on  them. 

Another  strategy,  which  is  based  on  the  design  of  cover 
stories  for  good  abstract  problems  has  been  introduced.  Problems 
of  controlling  the  generative  process  by  using  explicit 
specifications  and  the  problems  of  using  information  available 
regarding  the  student’s  familiarity  with  a  set  of  concepts  have 
also  been  briefly  dealt  with. 
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APPENDIX  Al  PROGRAnniNG  DETAILS 
L_  Languages  Used 

A!i  the  generators  mentioned  in  the  preceding  pages  uere 
programmed  in  the  list  processing  implementation  language  L*(F) 
(Newel  I,  et  al,  Jan.  1971).  The  main  implication  of  the  use  of 
L*(F)  is  that  one  adopts  a  specific  approach  to  system-building 
(Newell,  et  al.  Sept.  71).  This  approach  leads  to  the 
construction  of  large  systems  in  a  layered  fashion,  adding 
facility  after  facility  (e.g. an  association  mechanism  based  on 
the  use  of  hash-coding  techniques  for  accessing  association  lists; 
the  semantic  parser  discussed  in  Section  3.3)  in  a  number  of 
steps,  later  layers  using  the  tools  provided  by  the  earlier  ones 
and  building  on  them.  The  other  implication  of  the  use  of  (L*(F) 
is  that  the  total  access  provided  to  the  language  processor’s 
machinery  encourages  the  user  to  adopt  this  machinnery  uhen  needed 
rather  than  building  3  language  processor  on  his  own. 

Zju  P r  i  m  i  t  i  ve  opera  t  i  ons 

The  basic  operations  of  random  choice,  weighted  choice,  value 
assignment,  creation  of  associations  and  accessing  of  associations 
are  specified  by  the  use  of  the  operators  t  ,  ti  ?  and  8 
,  the  last  one  being  the  statement  term'n3tor  used  in  conjunction 
with  the  operators  «-  and  <=  .  These  six  operators  as  well  as  the 
comma  are  assigned  necessary  'character  actions’  and  'syntax 
actions  (Newell,  et  al,  Jan.  1371)  as  per  the  conventions  of 
L*(F),  ensuring  t.ie  proper  parsing  and  interpretation  of 
expressions  of  the  following  form  embedded  within  regular  l.Vr(F) 
pr-jgrams. 
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(A1  A2  A3  A4] 

(A1  A2  A3  A4]  t 

(A1  3  A2  5  A3]  t 

[R1  P2  R3]  t  X 


A  B  II 

ATT.  OBJ  <«  B  8 
ATT,  OBJ  ? 

ATT,  —  ? 


CoRifsi^n t  fields  follow  the 
semi -co I ons. 

Create  the  specified  data-list 
(of  type  list  on  L*(F)). 

Choose  an  item  randomly  from 
the  gi ven  I i st. 

Choose  by  a  weighted  random 
process.  Weights  are  prefixed 
with  the  number  sign. 

Choose  one  of  the  routines  in 
the  list  and  execute  it,  obtaining 
a  value.  Expressions  of  the  above 
forms  can  be  used  in  the  place  of 
the  variable  B  in  the  following 
statements. 

Replace  the  contents  of  list  A 
by  the  contents  of  list  B.  used 
mainly  to  assign  values  to  variables. 
Associates  a  value,  B,  with  the 
given  attribute  of  the  given  object. 
Obtains  the  value  of  the  given 
attribute  of  the  given  object. 

Obtains  the  value  of  the  given 
attribute  of  the  object  on  top 
of  the  Lft(F)  working  stack. 


The  brief  description  above  does  not  cover  all  primitives 
used  in  the  system.  However,  a  brief  mention  should  be  made  of 
variants  of  the  choice  operator  t.  One  has  a  memory  cell  for  its 
most  recent  choice  and  avoids  repeating  it.  Another  variant 
refers  to  information  associat ively  linked  to  its  data-list  by 
other  parts  of  the  program  before  making  a  selection.  This  fatter 
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variant  is  useful  in  programming  generators  which  are 
control lable. 

Si.  Creation  q±  Data-5tructures 

A  structure  synthesizing  routine  .C  is  used  extensively  by 
the  problem  generators.  The  execution  of  a  statement  of  the  form 

(ITYP,  ASUB1  VSUB1,  ASUB2,  VSUB2,  ASUB3  VSUB31  .C) 
creates  a  list  representing  a  token,  or  instance,  of  an  entity  of 
the  type  TYP.  Associat  ively  linked  to  this  list  are  the 
attributes  ASUB1 ,  ASUB2  and  ASUB3  which  are  assigned  to  values 
VSUB.I,  VSUB2  and  VSU83  respectively.  If  any  attribute  value  pair 
ASU8I  vSt)BI  is  followed  by  the  separator  .e  instead  of  the  comma, 
VSUBI  is  e-  ecuteo  (it  is  presumed  to  be  a  routine)  and  the  value 
obtained  is  assigned  to  the  attribute  ASUB1.  SIMILARLY  IF  TYP  is 
followed  by  ,E,  the  type  is  obtained  bj  executing  the  routine  TYP, 
the  type  is  obtained  by  executing  the  routine  TYP.  Examples  cf 
the  usage  follow: 

((LIST,  LINK-TYPE  SYMMETRICAL,  UNIQUE  TRUE!  .C) 
creates  a  structure  to  represent  'THE  SYMMETRICAL  LIST’. 

((ARRAY,  DIMENSION  2,  NUMBER  31  .Cl 
creates  a  structure  to  represent  ’THREE  TUG  DIMENSIONAL  ARRAYS'. 

There  are  special  separators  which  are  used  in  the  place  of 
the  comma  following  the  type  to  specify  ;n  a  compact  manner  the 
values  of  the  attributes  UNIQUE  and  NUMBER.  These  separators  are 
not  used  in  the  i  I  lustrations  in  this  Appendix  in  order  to  keep 
the  illustrations  simple. 

The  main  reason  for  the  utility  of  the  routine  .C  in  the 
programming  of  top-down  generators  is  that  it  can  be  used 
resursively.  A  specification  list  on  which  .C  operates  may 
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indicate  that  the  value  cf  ASUB1  should  be  computed  by  the  routine 
VSUBI .  Frequently,  VSUBi  itself  is  defined  in  terms  of  a 
construction  to  be  performed  by  .C,  creating  a  sub-structure  which 
ir  linked  to  the  main  structure  by  the  ASUBI  —  VSUBI  bond.  The 
following  example  illustrates  such  usage.  The  routine  TP 
specifies  the  construction  of  members  of  a  set  of  simple 
sentence-structures. 

TP:  ( I TP1  .E  OBJ.  TP2  .E  LOCATIVE  TP3  .E)  .Ci 

TP1 :  ( [FIND  DELETE  PRINT]  t 

TP2:  ( [NUMBER,  QUANT  ALL,  UNIQUE  TRUE,  NUMBER  0, 
r>DJC  TP21  .El  .C) 

TP21 : ( [ODD  EVEN  POSITIVE  NEGATIVE  PRIME  T) 

Assignment  of  *0’  to  NUMBER  is  interpreted  to  mean  that 
NUMBER>1.  The  routine  TP3,  not  defined  here,  accesses  a  fragment 
of  sentence-structure  assumed  to  hove  been  created  before  the 
execution  o*  the  statements  in  the  example.  The  accessed  fragment 
represents  an  object  such  as  ’THE  LIST  LI’,  ’THE  GIVEN  LIST*,  ’THE 
ARRAY  L (20) '  or  THE  ARRAY  L(20,  30)’.  TP3  synthesizes  a 

sentence- fragment  to  refer  to  the  object  or  ts  sub-objects.  The 
new  fragments  are  translatable  as  ’THE  FIRST  ROU  OF  THE  ARRAY’, 
’THE  ARRAY  L\  ’ALL  THE  SUBLISTS  OF  L’  or  ’THE  LAST  COLUMN  OF  L’. 

The  sentence  synthesizer,  briefly  described  in  Appendix  B, 
synthesizes  readable  sentences  hum  the  sentence- structures  of  the 
kind  produced  by  the  routine  TP,  to  generate  sentences  such  as  the 
f o I  lowing: 

FINO  ALL  THE  PRIME  NUMBERS  IN  THE  LAST  COLUMN  OF  THE  ARRAY  L. 

DELETE  ALL  THE  EVEN  NUMBERS  IN  THE  FIRST  SUBLIST  OF  L. 

PRINT  ALL  THE  POSITIVE  NUMBERS  IN  THE  ARRAY  L. 
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AF PEND IX  Bj_  IHE  SENTENCE  SYNTHESIZER 


The  sentence  synthesizer  deals  with  associative  structures 
representing  the  content  of  potential  sentences  or  parts  thereof. 
Fig.  7  showed  an  example  of  such  a  struct  re.  By  convention,  the 
sentence  structure  as  given  to  the  synthesizer  starts  with  the 
I  i st-structure  representing  an  occurrence  of  a  predicate  term. 
Linked  to  this  list-structure  by  labelled,  directed  associations 
are  sub-structures  representing  entities  that  stand  ir  various 
deep-case  relations  to  the  predicate.  Each  sub-structure 
similarly  consists  of  a  list-structure  representing  the  occurrence 
of  a  word-sense  with  associations  leading  off  to  related  entities. 
It  is  also  the  convention  to  bind  each  occurrence  of  a  word-sense 
with  relevant  markers  such  as  those  for  tense  number  and  gender 
using  attribute-value  associations. 

The  implementation  of  the  synthesizer  is  based  on  the 
principle  that  for  each  category  of  semantic  entity  in  the 
sentence  structure  there  should  be  a  synthesis  routine  which  has 
operational  knowledge  of  that  category  of  entity.  At  the  highest 
level,  the  synthesizer  finds  out  the  category  of  entity  it  has  to 
deal  with  and  triggers  off  the  associated  synthesis  routine. 
Recursive  calls  by  one  synthesis  routine  to  others  ere  necessary 
because  parts  of  the  sentence  structure  are  composed  of  smalle*- 
sub-structures. 

The  routine  dealing  with  predicates  looks  for  associations  to 
sub-structures  of  the  following  types:  subject;  voice  marker; 
tense  marker;  object,  in  case  of  transitive  verbs;  indirect 
objects;  locatives;  adverbial  of  purpose;  and  other  adverbial 
phrases.  The  routine  dealing  with  objects  attends  to  any  need  for 
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jterminers,  quantifiers,  adjectives,  number  markers,  and  several 
jpes  of  adjectives  and  adjectival  phrases. 
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